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Abstract 

Accident reports play a key role in the safety of complex 
systems. These reports present the recommendations that are 
intended to help avoid any recurrence of past failures. 
However, the value of these findings depends upon the causal 
analysis that helps to identify the reasons why an accident 
occurred. Various techniques have been developed to help 
investigators distinguish root causes from contributory factors 
and contextual information. This paper presents the results 
from a study into the individual differences that can arise 
when a group of investigators independently apply the same 
technique to identify the causes of an accident. This work is 
important if we are to increase the consistency and coherence 
of investigations following major accidents. 

1 Introduction 

Many different analytical techniques have been developed to 
support the identification of causal factors during accident 
investigations. Many of the proponents of these tools have 
argued that by using an agreed methodology, it is possible to 
increase the consistency of investigations [3]. In other words, 
similar causes will be identified for similar incidents. 
Accident analysis methods can also help to reduce individual 
bias; by encouraging investigators to follow and document 
common processes during an investigation [5], A small 
number of previous studies have been conducted to determine 
the veracity of these arguments, for instance as part of the 
Bielefeld Workshop series [1], 

This paper builds on previous work by presenting the results 
of an independent study involving three experienced analysts 
using common source materials, each applying the Safety 
through Organizational Learning (SOL) approach [7], The 
focus of the analysis was a collision into mountainous terrain 
that occurred in Alaska during 2010. Many other analysis 
techniques might have been used, including Why-Because 
Analysis, Systems-Theoretic Accident Model and Processes 
(STAMP), or Events and Causal Factors charting [3], SOL 
was chosen because it was intended to support the analysis of 


organisational and managerial factors in accident 
investigations. 

This paper focuses on the differences among individual 
analysts rather than the choice of particular techniques. It is 
concluded that the application of common analysis tools does 
not guarantee agreement over potential recommendations. In 
contrast to previous research that has stressed the importance 
of inter-analyst agreement; we would argue that such 
differences encourage teams of investigators to consider a 
range of interventions prior to the publication of an accident 
or incident report. 

2 Summary of the Accident Case Study 

The decision was taken to base our work on an existing 
accident report, rather than attempt to conduct the research 
using a ‘live’ investigation. This was justified for pragmatic 
reasons; it can be difficult to ensure access to an on-going 
investigation. There were ethical reasons: ensuring that the 
findings of our analysis do not bias the recommendations that 
are essential to avoid the repetition of previous fatal 
accidents. This paper focuses on the loss of a single -engine, 
turbine -powered, amphibious float-equipped de Havilland 
DHC-3T. The pilot and four passengers received fatal 
injuries. Four other passengers received serious injuries. The 
flight took off from a private lodge on the shore of Lake 
Nerka at 14:27 towards a remote sport fishing camp about 52 
nm southeast on the Nushagak River. The accident pilot was 
familiar with the route. His aircraft was equipped with two 
global positioning system (GPS) units with moving map and 
terrain display capabilities and a radar altimeter with visual 
annunciator and aural tone capabilities. The NTSB found that 
in spite of these navigational systems, the aircraft diverted 
from the expected course during the final minutes of the 
flight. It headed towards mountainous terrain and eventually 
crashed. 

The investigation examined a range of human factors 
concerns that might have explained the pilot’s deviation from 
the expected heading. These included fatigue as well as 
‘major life events’; including his retirement and the sudden 
death of a family member. The NTSB found that he had 
suffered an intracerebral hemorrhage four years before the 
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accident. It was concluded that the probable cause was the 
pilot’s temporary unresponsiveness ‘for reasons that could not 
be established from the available information’. A number of 
recommendations were made in the final report [6], In 
particular, they focussed on inadequate regulatory guidance 
for the medical certification of aircrew. 

3 Introduction to SOL 

Safety through Organizational Learning (SOL) is a general 
approach to analysing adverse events [2] . It has been widely 
used across many industries in several different countries and 
is one of a number of analytical tools and techniques that can 
be used to assist accident investigations [1, 3]. The popularity 
and longevity of the approach make it an appropriate starting 
point for the identification of individual differences in the 
application of common approaches to mishap investigation. 
A further justification is that SOL emerged from several 
different strands of research within the area of ‘socio- 
technical systems analysis’. This is important because 
managerial and organisational factors are often the most 
complex to investigate and at the same time are amongst the 
most significant causes of incidents and accidents. SOL 
analysis considers the role of technology, individuals, teams, 
organisations and the environment. By considering this range 
of potential causes, analysts are encouraged to look beyond 
individual ‘root cause’. The identification of the five 
common sub-systems arguably increases consistency between 
investigators - they must consider the same five areas of 
concern. The SOL analysis technique is based around a 
number of different stages: 

1. Identify the events that occurred during the accident; 

2. Organise these events into Event Building Blocks, 
these record the time, location, actor(s) involved and 
the action(s) that contributed to an incident or 
accident; 

3. Arrange the Event Building Blocks into a Time- 
Actor Diagram, this provides a time-line of events 
leading to an accident; 

4. Identify contributing factors for each Event Building 
Block in the Time-Actor Diagram. 

The direct and indirect factors associated with the Event 
Building Blocks include: 

A. Representation of Information; 

B. Communication; 

C. Working conditions; 

D. Personal performance; 

E. Violation; 

F. Technical components. 

The SOL technique also considers a range of indirect factors 
including control and supervision; group influences; rules, 
procedures, and documentation etc. The following sections 


summarise the results that were obtained for three different 
investigators each applying the SOL analytical tools to the 
collision into mountainous terrain summarised in the opening 
sections of this paper. The intention is to illustrate the 
subjective differences that arise during the application of 
common analytical tools. Most of the attention is on the 
development of Time-Actor diagrams, which arguably lies at 
the heart of the SOL approach. The decision to focus on 
these artefacts is also justified by the difficulty of 
summarising the results without extending the descriptions 
across many pages of prose. 

All three investigators have a similar background in terms of 
expertise and experience in the application of accident and 
incident investigation techniques, representing a cumulative 
total of more than 40 years of experience in the application of 
causal analysis tools. However, this expertise has been 
obtained across a range of investigations including software 
systems analysis, aviation applications and maritime accident 
investigations. 

4 Analyst A 

Figure 1 illustrates the results from an initial SOL analysis by 
the first analyst. It presents a Time-Actor diagram for the 
decision to issue the accident pilot with the medical certificate 
required to fly after his intracerebral hemorrhage (ICH) 
during March 2006. He made repeated attempts to regain 
certification. Under 14 CFR 67.109, “a transient loss of 
control of nervous system function(s) without satisfactory 
medical explanation of the cause” prevents pilots from 
gaining a medical certificate. 14 CFR 67.401 states that a 
medical certificate can be granted under special 
circumstances to a person who does not meet the regulatory 
provisions if they satisfy the Federal Air Surgeon or delegated 
authority that they can fulfil the associated duties without 
endangering public safety. The decision to allow the pilot to 
regain their medical certificate was also guided by the FAA’s 
internal Aeromedical Certification Reference Manual; 
“special issuance consideration will be given to those who 
can demonstrate full recovery of motor, sensory, language, 
and intellectual function.” The Alaska Regional Flight 
Surgeon’s decision is shown as the bottom event in Figure 1. 

Figure 2 extends the Time-actor diagram beyond the initial 
decision to re-issue the pilots medical certificate after he 
recovered from the ICH. In particular, it denotes the possible 
impact of his recent bereavement and the observations by co- 
workers that this tragedy might have had upon the pilot. As 
mentioned above, subsequent stages of the SOL analysis 
build on these diagrams to identify a range of direct and 
indirect factors. Analyst A focused on communication issues 
and on regulatory concerns that led to the certification of the 
pilot, illustrated in Figure 1. In contrast, the following 
sections present interim results from the application of SOL 
by the remaining participants in the study. 
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ACTORS 


Time: March 
22, 2006 


Pilot of 
DHC-3T 


Suffers 

intracerebral 

hemorrhage 

(ICH) 


Neurological 

specialist 


Time: July 
21, 2006, 

Repeat MRI 
negative for 
any changes 


Time: March 26, 2007, 
Follow-up outpatient 
neurology evaluation - 
no strokes, seizures, 
vision changes, 
significant changes to 
ambulation or gait, or 
other difficulties since 
Pilot last seen after ICH. 


FAA 


Local 

Neurologist 


Alaska 

Regional 

Flight 

Surgeon 


Time: March 26, 


Time: April 8, 2008 

2008 medical 


Medical certificate 

certificate not issued, 


issued with warning. 

deferred for further 



FAA evaluation 




Time: March 26 2007 


Time: May 7, 

Airman medical 


2007, 

certificate not issued and 


Concludes pilot 

was deferred for further 


needs two year 

FAA evaluation 


recovery period. 


Time: March 3, 2008. 
Concludes “appears 
entirely normal. I see no 
neurological deficits 
whatsoever”. 


Time: April 8, 2008 

Primarily used FAA Aeromedical 
Certification Reference Manual to 
guide his determination, variety of 
FAA reference materials provide 
general guidance but still require 
application of clinical judgment. 


Figure 1: SOL Time- Actor Diagram for Analyst A, Focussing on FAA Certification of Pilot 


ACTORS 


Pilot of 
DHC-3T 


Time: July 29, 

2010, 

Pilot takes 
leave following 
sudden death of 
his son-in-law 
and returned on 
August 4, 2010 


Time: 09:02 

Flies to 
Dillingham 
Airport (DLG), 
dropped off 
another pilot 
and then 
returned to the 
lodge at 11:20. 


Time: 14:27 

Departs private 
lodge on Lake 
Nerka for 
remote sport 
fishing camp 
about 52 nm SE 
on Nushagak 
River. 


Time: 14:28 

Pilot uses 
different 
direction during 
takeoff to avoid 
“wind and 


Time: 14:42 

De Havilland 
DHC-3T 
impacts 
mountainous, 
terrain 10 nm 
northeast of 
Aleknagik, 
Alaska. 


Lodge 

Manager 


Lodge 

Employee 


Time: July 29, 2010++, 

No overt change in the 
accident pilot’s behavior 
following his son-in-law’s 
death. 


Time: July 29, 2010++, 

Spoke to accident 
pilot about his son-in- 
law’s accident, stated he 
was worried about his 
daughter and her three 
children. 


Figure 2: SOL Time-Actor Diagram for Analyst A, Immediate Events Prior to Collision With Mountainous Terrain 
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5 Analyst B 

Following the stages of the SOL analysis technique described 
in the introduction, analyst B began by identifying the events 
during the accident, i.e. what happened. An event is 
determined by a sequence of singular actions by different 
actors (maybe a person or a technical component) between a 
starting point and an end point. The starting point is defined 
as the first alarm or the first perceived deviation from a 
warranted course or action. According to the SOL guidelines 
the situational description should focus on observable facts 
only concerning what happened, without hypothesis about 
causes or why it happened [6], After building the chain of 
events. Analyst B identified contributing factors. This aspect 
of SOL forms a strong contrast with other approaches, 
including STAMP, that deliberately abstract away from the 
chain of events leading to an accident [3], Further work is 
required to determine whether these alternate approaches 
offer greater or lesser degrees of consistency when applied by 
different investigators. 

The work of all three analysts in building the Time-Actor 
diagrams that represent accident events was hindered by a 
lack of evidence following this accident. The pilot received 
fatal injuries when the airplane impacted the mountains and 
could not, therefore, provide direct evidence about the course 
of the accident. The witness statements from the four 
surviving passengers were of limited value; it was difficult for 


them to form a clear view of the events and other factors 
influencing the pilot’s behaviour. The airplane did not have a 
flight recorder system installed. 

As with Analyst A, the focus of the second investigators work 
also settled on the pilot’s intracerebral hemorrhage (ICH) and 
the subsequent decision to reissue his aviation certificate after 
medical and neurological evaluations. The results from the 
evaluation indicated that the accident pilot had fully 
recovered from the ICH, and that he did not have any 
neurological deficits. The Federal Aviation Administration 
indicated that they at the time had issued a total of 19 first- 
class airman medical certificates to pilots following ICH. 
However, Analyst B focussed more directly on the possibility 
that the accident pilot had experienced a medical condition 
leading to transient incapacitation or impairment, as a 
contributing factor to the accident. An autopsy performed on 
the accident pilot did not identify any direct evidence of such 
a problem. Two independent neurologists confirmed the lack 
of evidence that the accident pilot experienced a specific 
medical event or condition during the accident flight. This 
contrasts with the first analysis, which placed greater weight 
on the previous medical history than on the absence of direct 
medical evidence for transient incapacitation or impairment 
following the crash. This difference of emphasis is important 
because it typifies the conflicts that can arise in accident 
teams where there may often be insufficient or contradictory 
evidence available in complex investigations. 


Time: 1442 
Impact with mountain 
when the airplane 
was in a climbing left 
turn 



Time: 1438:59 
Last position record 
received from Sky 
Connect 

i i 

■ i 

■ i 

■ i 

■ i 

i i 

i i 

i i 

■ i 

i i 

Time: 4-6 sec before impact 
Pilot receives aural and visual 
altitude alerts from the airplane’s 
radar altimeter system 


♦ 



Figure 3: SOL Time- Actor Diagram for Analyst B Illustrating the Lack of Evidence about the Causes of the Accident 


Figure 3 illustrates the results from Analyst B’s application of 
the SOL technique. As the factual investigation did not 
establish any exact evidence to determine what actually 
happened in the critical minutes before impact. Analyst B 
regarded the causes as entirely speculative. The lack of 
observable facts leading up to the accident is represented by 
an empty event block in the diagram. This is not a standard 
part of the SOL technique. However, the use of such 
notational extensions is common in investigatory boards 
where local adaptations are often encouraged to meet the 
demands of particular accidents. However, they also illustrate 


the potential for further inconsistency when others do not 
exploit the same notational adaptations used by their 
colleagues. 

Analyst B focused on lack of observable facts about events 
leading to the accident, the first alarm or the first perceived 
deviation from a warranted course or action. In this case the 
accident pilot’s actions in the 3-minute period between the 
airplane’s last reported position and the time of impact 
remains unknown. However, this lack of evidence raised 
further questions for Analyst B. The absence of observable 
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facts stems from a host of regulatory and organizational 
concerns. Valuable information about the accident pilot’s last 
actions could have been provided if a flight recorder system 
was installed in the airplane. 

In 2003 the National Transportation Safety Board (NTSB) 
issued a safety recommendation asking the Federal Aviation 
Administration (FAA) to require that such equipment be 
installed on aircraft like the accident airplane. The FAA, 
however, did not implement this recommendation. This 
decision made by the FAA is regarded as the indirectly 
contributing factor, which impedes organizational learning in 
order to enhance safety and prevent future accidents. Again, 
this focus reflects further differences between Analyst A and 
Analyst B. By expanding the scope of the investigation to 
question the lack of evidence, this second application of SOL 
illustrates the benefits to be derived from allowing multiple 
investigators to apply a similar toolset to the same accident 
[2], The individual differences between investigators, 
identified by Lekberg [5], survive the application of tools and 
techniques that have been developed to ensure the consistency 
of causal analysis. 

6 Analyst C 

The third investigator had previously used a number of 
different analytic techniques for casual factor identification. 
However, this case study was their first attempt to use SOL. 
The application of causal analysis techniques can raise a host 
of problems; many approaches lack detailed technical 
documentation and training material. A number of 
documents provide case studies in the application of SOL [2, 
6], It can still be difficult to apply the concepts and ideas to 
new accidents and incidents. It is for these reasons that 
Analyst C’s initial time-actor diagram was relatively simple, 
as shown in Figure 4. They justified this sketch using similar 
arguments to those introduced by the second investigator. 
They believed that no other pertinent and certain information 
could be added to the Time-Actor diagram in Figure 4, based 
on the evidence that was gathered in the aftermath of the 
accident. 


ACTORS 


Time: Aug 9, 


Time: Aug 9. 

2010. 1427 


2010. 1442 

Takes off from 


Flight impacts 

private lodge or 


mountainous 

remote sport 


terrain near 

fishing camp 


Aleknagik, 

about 52 nm 


Alaska 

away. 



Figure 4: Initial SOL Time-Actor Diagram for Analyst C 

The lack of evidence about the immediate causes of the 
accident led Analyst C to create a new time-actor diagram 
focussing on the NTSB’s inability to determine why the pilot 
was temporarily unresponsive. This diagram is shown in 


Figure 5. As can be seen, it builds on many of the concerns 
identified by Analyst B as a result of their findings derived 
from Figure 3. Figure 5 begins nearly seven years before the 
crash, when the NTSB first recommended that the FAA 
require aircraft such as the accident aircraft be equipped with 
flight recorders and proceed through similar 

recommendations. The diagram also focuses on the FAA’s 
decision to not implement these recommendations. Much 
more could be said about the regulatory, organisational and 
governmental pressures that influence the different decisions 
and recommendations made in the aftermath of this and 
similar accidents. Brevity prevents this more sustained 

presentation. It is only possible to provide an outline of the 
individual differences between the investigators in a 
conference paper. However, a journal paper is presently 
being developed to document the extended results from our 
comparisons. 


7 Comparisons and Analysis 

Previous sections have presented the results from three 
different analysts applying SOL to the same incident. The 
following findings can be identified from our initial study. 

Differences in the scope of the analysis - the first investigator 
focussed on certification of the accident pilot. In contrast, the 
remaining two analysts argued that there was no direct 
evidence of the pilot being incapacitated and instead looked at 
the reasons why there was insufficient evidence about the 
events leading to the mishap. 

Differences in interpreting absent information - the 
investigators took different approaches to the lack of evidence 
about the causes of the accident. Analysts B and C took a 
strict view; they restricted their analysis to those events for 
which there is direct evidence. It can be argued that to go 
beyond the available evidence is to risk speculating about the 
cause of an accident. In contrast. Analyst A argued that the 
accident provided ‘learning opportunities’ from apparent 
weaknesses in the medical re-certification of aircrew even if 
there was no evidence to suggest that the pilot had been 
incapacitated in this accident. 

Differences in the Use of the SOL Notation - the three 
analysts worked independently during the study. Partly in 
consequence, it is possible to see a number of differences in 
their use of the Time-Actor diagrams. These include the level 
of detail, for example between Figures 2 and 4. The analysts 
also introduced new features to help them model aspects of 
this accident, including the blank event in Figure 3 by Analyst 
B to denote the lack of evidence. As mentioned, these 
changes increase the flexibility of the SOL approach in order 
to represent and reason about particular mishaps. However, 
they can undermine consistency that might otherwise enable 
wider comparisons between the analyses of different 
accidents by different investigatory agencies using SOL. 

Differences in Confidence and Experience Using SOL -after 
the study, the three investigators reported different levels of 
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confidence in their application of the technique. This can be 
explained by different levels of expertise in applying the 
technique and also by the continuing need to support causal 
analysis techniques with appropriate training and 

documentation [4]. 

Differences between Alternate Techniques - this study 
arguably raised more questions than it answered. In 
particular, it remains to be seen whether the independent 


application of different techniques by a groups of 
investigators working on the same incident would also yield 
differences similar to those summarised in this section. Other 
questions relate to international differences between 
investigators. This is particularly important given that global 
market have led to mishaps being repeated in different 
countries [3]. This paper and the Bielefeld workshops have 
provided some insights into these topics [1]; however, much 
remains to be done. 


ACTORS 


Pilot of 
DHC-3T 


NTSB 


Time: Dec 22, 
2003 

Issued Safety 
Recommendation 
A-03-64 asking 
FAA to require 
crash-protected 
flight recorders on 
aircraft like the 
accident airplane 
by January 1, 

2007 


FAA 


Time: Aug 9, 
2010. 1442 

Flight impacts 
mountainous 
terrain near 
Aleknagik, 
Alaska 


Time: Feb 9, 2009 

Classified A-03-64 
“Closed- 
Unacceptable 
Action/Superseded 
and issued A-09-10 
and A-09-1 1 asking 
the FAA to require 
crash-resistant flight 
recorder systems on 
all “existing turbine- 
powered ... aircraft 
that are not equipped 
with a [CVR / 

FDR]” 


Time: Aug 9, 


Time: Dec 


Time: February 


Time: May 24. 

2010 


23, 2010 


201 1++ 


2011 

Began 


Classified A- 


A-09-10 and A- 


Adopted report 

investigation 


09-10 and A- 


09-1 1 remain 


stating that 

into the 


09- 1 1 “Open- 


classified as 


“reasons . . . could 

accident 


Unacceptable 


“Open- 


not be 



Response” 


Unacceptable 


established” for 




Response” 


the pilot’s 










unresponsiveness” 


Time: February 15, 
2011 

Notified the NTSB 
“that it did not intend 
to mandate the 
equipage of additional 
recording systems . . . 
as recommended.” 


Figure 5: SOL Time- Actor Diagram for Analyst C 

8 Conclusions and Further Work 

Accident reports preserve safety by presenting 
recommendations that can help avoid any recurrence of past 
failures. The value of these findings depends upon the causal 
analysis that explains why an accident occurred. This paper 
has identified some of the differences that arise when 
investigators independently apply an analysis technique to the 
same accident. The results show that different investigators 
focus on different aspects of an accident; this confirms 
previous work by Lekburg [5], Our work also revealed 
differences in the interpretation of missing evidence, 
differences in the use of common notations and differences in 
the level of confidence in the approach. A key finding from 
this study has been to question the way in which previous 
studies emphasise the benefits of causal analysis tools in 
enhancing consistency [3]. Not only can it be shown that 
individual differences persist in the use of these techniques 
but it can also be argued that this is a strength; encouraging 
teams to explore a wide range of lessons from previous 
accidents. 
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